Job Migration

A Databricks job is a task graph plus a pile of notebooks and Python files, with schedules, conditions, loops, and email notifications wrapped around it. Phase 3 of our migration utility takes that whole shape and turns it into two things: files that run on Yeedu, and an Airflow DAG that runs them in the right order.

You give it a list of job IDs. It gives you a verdict on each one.

What comes in, what goes out

Databricks	Yeedu
notebook_task	A Yeedu notebook, uploaded to a Yeedu workspace
spark_python_task / python_task	A Python file in the same workspace
Task dependency graph	Airflow task dependencies in a generated DAG
for_each_task	A mapped task, expressed with YeeduOperator.partial(...)
condition_task	A BranchPythonOperator
Quartz-style schedule	QuartzTimetable
Email notifications	EmailNotificationHook from the Yeedu Airflow provider
Job run	YeeduOperator, from yeedu.operators.yeedu

Task types outside that list, spark_jar_task, sql_task, pipeline_task, dbt_task, run_job_task, and python_wheel_task among them, aren't converted. The job is reported as having an unsupported task type and skipped rather than half-migrated.

warning

File paths starting with /dbfs/ are ignored and never downloaded. Yeedu has no DBFS. Move those files to workspace or object-storage paths before you migrate.

What converts automatically

Each downloaded file goes through a chain of processors that rewrite it in place. This is the part that saves the most human time, and it's worth knowing exactly what it touches.

Rewrite	What happens
DeltaTable to YeeduDeltaTable	Every non-comment occurrence of the identifier is replaced, with a # Changed to make it work in Yeedu comment inserted above the line
from delta.tables import DeltaTable	Commented out
hive_metastore to your target catalog	Case-insensitive whole-word replacement with TARGET_UNITY_CATALOG_NAME
OPTIMIZE, VACUUM, CACHE TABLE	Table references are wrapped in backticks for compatibility, including inside f-string SQL
Table modification statements	INSERT, UPDATE, DELETE, DROP, ALTER, and TRUNCATE targets are parsed so the tables they write can be identified. SQL is parsed with an AST, falling back to regex when parsing fails
Dry-run suffixing	Target tables and storage paths get a ___yeedu suffix. Double suffixes collapse back to one
dbutils calls	Logged to dbutils_usage.csv. dbutils.notebook.run targets are additionally recorded as notebook references, so the DAG generator can wire the dependency
Identity-bound functions	current_user(), session_user(), is_member(), and is_account_group_member() are detected and recorded as OAuth requirements

Rewrites apply to the downloaded copies under the run directory. Your Databricks workspace is never modified.

What does not convert automatically

Permissions, mostly. The utility identifies every Unity Catalog table and view your code references, walks the full dependency chain for each using view_edges.csv, then checks whether the target principal holds USE_CATALOG, USE_SCHEMA, and SELECT on all of it, and it does the same for the ADLS Gen2 storage underneath, caching validation results per storage account and container so that a hundred tables in one container cost one check rather than a hundred. That dependency walk is the part worth paying attention to, because a view three levels deep over a table nobody remembers granting on is the single most common reason a migrated job fails on its first production run, and finding it during analysis costs nothing while finding it at 3am costs a night.

Where a grant is missing, we don't guess. We write the GRANT statements into notebooks you run in Databricks yourself, split by level so you can decide how coarse to be.

Notebook	Level
required_grants/01_catalog_grants.ipynb	Catalog
required_grants/02_schema_grants.ipynb	Schema
required_grants/03_object_grants.ipynb	Table and view, consolidated

Granting at schema level is easier to live with than granting per object. Setting AUTO_RUN_REQUIRED_GRANTS=true runs the schema-grants notebook and re-validates in one pass.

Beyond permissions, three things stay yours: DBFS paths, mixed-language notebooks, and any semantic difference in how current_user() resolves under a new identity. See What Is Supported.

The readiness verdict

Every job comes out of the analysis marked READY or not, in job_readiness_report.csv, with a specific reason attached. A job is READY only when none of these produce a finding.

Check	Blocks readiness when
Job processor status	The job was already marked FAILED, because a file couldn't be downloaded or a task type is unsupported
Table access	The principal lacks USE_CATALOG, USE_SCHEMA, or SELECT on a referenced object or one of its dependencies, or the ADLS location behind it is unreachable
dbutils usage	The code calls a dbutils method outside the analyser's supported set. The reason reads Unsupported dbutils method: <method>
Python UDF dependencies	The code depends on a Python UDF registered in Unity Catalog
OAuth requirements	The code uses an identity-bound function such as current_user() and OAuth credentials aren't configured
Miscellaneous unsupported operations	SHOW EXTERNAL LOCATIONS or SHOW GRANTS ON EXTERNAL LOCATIONS appears in the code

The dbutils check runs against a fixed allowlist that is slightly narrower than what our runtime actually implements. What Is Supported names the calls that work anyway.

Dry runs

Nobody should point a freshly converted job at production tables on its first execution, so we built the alternative in.

Set APPEND_YEEDU_SUFFIX=true and every table written to and every storage path picks up a ___yeedu suffix. The utility also generates a notebook that creates those test tables, populated with a limited row sample, and another that drops them again. Set AUTO_CREATE_DRY_RUN_TABLES=true and it runs the create notebook for you.

Run the suffixed jobs, compare the output against the Databricks originals, then run for real.

What a run leaves behind

Each run writes a timestamped folder.

run/run_20231027_103000/
├── airflow_dags/          # Generated Airflow DAG Python files
├── databricks_files/      # Downloaded notebooks and scripts, modified in place
├── observations/          # CSV reports detailing findings
│   ├── dbutils_usage.csv
│   ├── distinct_objects.csv
│   ├── file_table_lineage.csv
│   ├── jobs_summary.csv
│   ├── oauth_requirements.csv
│   ├── patterns_found.csv
│   └── processed_files.csv
├── required_grants/       # Notebooks with GRANT statements for missing permissions
├── sql_scripts/           # Generated SQL scripts for dry-run testing
└── job_readiness_report.csv

Start with ready_to_run.csv, which is the subset of jobs that passed every check, then read job_readiness_report.csv for the ones that didn't. observations/file_table_lineage.csv tells you which file references which object and whether access validation passed, which is usually the fastest route from a failed job to the grant it's missing.

Getting it running

The utility processes jobs in parallel, controlled by MAX_PARALLEL_JOBS_TO_PROCESS, and is configured entirely through environment variables. Three are mandatory: DATABRICKS_HOST, DATABRICKS_TOKEN, and UTILITY_BASE_PATH.

Two entry points exist. 03_databricks_to_yeedu/migration_wizard.ipynb is the supported one, and it fills in the environment through notebook widgets. 03_databricks_to_yeedu/main.py runs from a shell, taking --job_ids as a comma-separated list or the literal all.

python main.py --job_ids 12345,67890 --log_level DEBUG

Optionally, the utility pushes its output the rest of the way: YEEDU_PUSH_TO_WORKSPACE=true creates a Yeedu workspace and uploads the processed files as notebooks, and YEEDU_PUSH_DAGS_TO_AIRFLOW=true uploads the generated DAGs through your Airflow instance's Code Editor API. By default only DAGs for READY jobs are pushed.

Running main.py --help prints the full environment-variable reference, which is the authoritative list. It's a long one, covering Yeedu workspace targets, Airflow credentials, OAuth credentials for identity-function validation, and the dry-run switches.

After the analysis

1. Read ready_to_run.csv and job_readiness_report.csv.
2. Run the notebooks in required_grants/ in Databricks to close the permission gaps.
3. Fill in the service principal details in adls_credentials.csv, with the client secret encrypted. See Migration Utility.
4. Run the first migrated jobs on Yeedu and compare results against the original Databricks runs.

Step four is the one that matters. Everything before it is preparation.